Papers with length generalization
Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought prompting is a powerful technique for enhancing language model’s reasoning capabilities, but generating long and correct CoT trajectories is challenging. |
| Approach: | They propose to align the steps of Chain-of-Thought reasoning with loop iterations and apply intermediate supervision during the training of Looped Transformers. |
| Outcome: | The proposed method generates accurate reasoning chains for complex problems exceeding training length, and improves performance of the auto-regressive model. |
Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Tasks (2025.acl-long)
Copied to clipboard
| Challenge: | Existing models excel in arithmetic reasoning but their generalization capabilities are incompletely understood. |
| Approach: | They propose a theoretical framework for understanding the generalization behaviors of transformers in arithmetic tasks, focusing on length generalization. |
| Outcome: | The proposed framework can predict generalization behaviors in transformers with a high translation invariance and base mismatch in modular operations. |
Learning to Reason from Feedback at Test-Time (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to utilizing feedback are expensive and lack the time to perform iterative interactions with the environment. |
| Approach: | They propose a novel paradigm that formulates feedback utilization as an optimization problem at test time and a learnable test-time optimizer to effectively exploit feedback. |
| Outcome: | The proposed paradigm improves scalability and performance on two large language models across four reasoning datasets. |
Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | a recent study explores efficient ultra-long context modeling. |
| Approach: | They propose to use Hierarchical Sparse Attention to achieve efficient ultra-long context modeling. |
| Outcome: | The proposed model performs comparable to full-attention baselines on in-domain and out-of-domain tasks. |
How do autoregressive transformers solve full addition? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large pre-trained language models have demonstrated impressive capabilities, but there is still much to learn about how they operate. |
| Approach: | They investigate the ability of the autoregressive transformer to perform basic addition operations by using causal analysis to find that a few different attention heads in the middle layers control the addition carry . they found that due to the lack of global focus on the sequence within these attention heads, the model struggles to handle long-sequence addition tasks. |
| Outcome: | The model performs basic addition tasks, but it still faces challenges with length generalization. |
Bridging the Memorization-Utilization Gap: Near-Lossless Context Compression via Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in context compression have failed to effectively utilize compressed representations for downstream tasks. |
| Approach: | They propose a holistic training paradigm that uses outcome-based RL to enable implicit expansion. |
| Outcome: | The proposed model outperforms previous models on NIAH, LongBench and multi-hop reasoning. |